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ABSTRACT 

This exploration of evaluation strategies for 
systemic educational reform considers whether there is a way to 
design an assessment and delivery system that can accomplish the 
goals of the total educational process. A basic question that must be 
addressed in systemic reform is the role of education in the 
socialization processes of society. Beyond the common desire to 
produce an informed and educated citizenry, goals become less clear. 
Some goals and conflicts are considered in the context of the 
community college. The American Association of Community Colleges has 
identified a core of 13 indicators that can be used to assess college 
effectiveness. Evaluating college effectiveness from the perspectives 
of students and faculty may result in a move toward the pragmatic and 
a shift in systemic evaluation to value social as well as academic 
priorities. Systemic reform in this instance might require asking how 
di s c ip 1 ine~spec i f i c knowledge fits in with the larger view of 
education’s role in the development of the whole. The paper takes the 
position that a delivery system should be created that emphasizes 
what people should be able to do when they are finished with what the 
system provides. Before this can be accomplished, it will be 
necessary to understand what education should be and its desired 
outcomes. Reform must be more than polishing the brass on a ship that 
is sinking. (Contains seven references.) (SLD) 
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Assessment and Education Reform: 

Doing More Than Polishing Brass on the Titanic, 
a Call for Discussion 



When Norman asked me if I was interested in participating in this conference to 
address evaluation strategies for systemic reform, I was intrigued by the opportunity. 
Initially the subject for the conference seemed straight forward--what questions should 
we as institutional representatives ask in order to provide insight and guidance to the 
process of instructional evaluation and systemic reform. My involvement in the 
educational process, apart from an occasional repass at the university, has been working 
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in community colleges. I therefore felt that I could provide an alternative perspective, 
particularly in how systemic reform is affecting the community college and visa versa. 

Certainly this is not a new question. As educators we deal with assessment and 
outcome evaluation each time we enter the classroom or develop a means for measuring 
students’ grasp of class content. Administratively we address effectiveness at each 
advisory committee meeting and throughout the year in the process of program review 
and strategic planning. An “ Evaluation Strategies Working Group” — I found myself 
rolling the terms around and around in my head, trying to crystallize what the desired 
outcomes might be from such a group. The objective was basically clear— to formulate 
questions that can be posited to the group, which would help initiate discussion about 
assessment of systemic reform and outcome evaluation in education. However, I found 
that the more information I received from Norman about the NISE evaluation conference, 
and the more I discussed it with faculty and administration at Mt. Hood Community 
College, the more convoluted my thinking became. 

The issues raised began to suggest the complicated nature of the problem we have 
chosen to explore. Given the diversity of institutions delivering the education in the K 
through Graduate School system— residential and commuter colleges, private and public 
sector, distance learning and e-mail colleges, is there a way to design an assessment and 
delivery system which accomplishes the overall goals of the total educational process? 
Can these disparate processes consummate in a product (graduate or participant) with 
such common qualities that it would be difficult to tell in which arrangement of delivery 
systems the individual participated? How do these various opportunities for learning and 




4 



socialization affect the students’ ability to participate positively in some capacity? Given 
that academic competency is part of the total socialization process integral to the 
educational system, do each of these meet that need equally well? 

The National Education Goals Panel charges education to make it possible that 
“by the year 2000, every adult American will be literate and possess the knowledge and 
skills necessary in a global economy and exercise the rights and responsibilities of 
citizenship.” If a desired outcome of the system is to produce a functioning member of a 
cohesive society, then we need to create the systemic environment which ultimately leads 
to that result. The fabric of our society is becoming ever more loosely knit. We no longer 
need to communicate face to face, or even talk to each other, and people on the whole are 
becoming increasingly more paranoid and suspicious of their fellow man. Technology 
shifts continuously diminish the need or opportunity; to interact with each other. We 
commute in individual vehicles, receive money automatically deposited in our accounts, 
withdraw needed funds from a machine, and pay our bills by mail. A basic question that 
must be addressed in systemic reform of education is what is the role of education in the 
socialization processes of society. 

Distance learning which enables students to avoid classrooms and all the rest of 
the social commitments and interactions necessary to participate in a classroom, may 
contribute to the atomization of our society. Is education actually contributing to the 
further decay of societal skills and the abilities of individuals to function in a larger part 
within society, or is the system really meeting these needs by embracing the 
technological shift. Phone registration, payment with credit card, classes via television, 
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communication through e-mail, diploma received by mail— are these exemplary models 
for our society? This is not meant to question the value of learning for its own sake, but 
rather to broach the issue of the role of the educational system in encouraging an ever 
more atomized society, unschooled in the human aspects of living and working together. 
When we talk about systemic reform we are addressing the idea of the vision of the world 
and how we would like it to be. If what we are trying to accomplish is to create a 
cohesive society comprised of literate and social individuals, the movement of education 
reformation will/must include an aspect of recognition of that obligation. 

Most of the people with whom I spoke were comfortable with the idea that a 
major goal of the academic system in America is to produce literate citizens who possess 
the ability to synthesize information in a way that allows them to make informed 
decisions, regardless of whether or not that decision is in the context of family, daily 
living, work or politics. Further it seems to be accepted that the institution of education 
plays a role in the socialization processes of our citizens in the morays and values of our 
democratic society. Beyond this common denominator of a desire to produce an 
informed, educated citizenry, things become more vague. The question relating to what 
specifically is most important for graduates to obtain from the system, what skills are 
most consequential in our quest to achieve the goal of an educated citizenry, varies 
depending upon whom one asks. Consequently, I decided to examine the question from 
several perspectives each of which posits questions relating to the process of assessment 
and systemic reform. These perspectives include: 

• 




the student and (science) faculty members, 
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• the community college’s role in community and regional development, and 

• systemic reform, the “umbrella” result(s) 

Due to the interdisciplinary nature of the system being assessed and the multiple 
outcomes, none of the areas are freestanding. The internal cycle of assessment, system 
reform, assessment, is dynamic and takes place at each level as well as between levels, as 
one area makes change and the other areas respond and make adjustment. I have for 
purposes of this paper decided to address them in reverse order, going from the broadest 
theoretical realm to pragmatic application of the faculty and their assessment of how what 
they are doing in their individual classrooms affect achieving systemic reform. This is 
followed by a review and further discussion section, which revisits and expands upon the 
major points with an exploration of issues and processes of systemic reform. 

Systemic Reform 

How do outcomes connect to systemic reform? I recently discussed assessment 
and systemic reform with several faculty members. In our discussion they had difficulty 
not getting caught up in the chicken or the egg conundrum. Which do we do first— reform 
the process or reform the outcomes? It seems logical to formulate the outcomes and then 
move in on the process to achieve the desired results. The working hypothesis is that once 
you know what you want in the end, you can reform what is in place to provide the 
setting for the student to accomplish that goal. The assumption is that systems function 
most efficiently when goal are clear, support is comprehensive, and expectations are 
realistically high. 
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We need the vision to know where we are going, and what it is that we wish to 
achieve in order to attain our goals, or even recognize when we arrive. In words attributed 
to Yogi Berra, “you’ve got to be very careful if you don’t know where you are going, 
because you may not get there.” How do we know what we should be doing? Why it is 
important to accomplish a particular outcome? In order to answer these questions we 
must have a clear understanding of the “big picture” outcomes. As a national system, 
provisioning education to the masses, how do we know what we should be doing? What 
are the indicators which help us know if we are in fact being effective at what we believe 
we should be accomplishing? 

What we measure, and how we measure changes for each region, district, 
educational domain (university, community college, K-12), and institution differs by 
who, or what agency or individual is assessing the effectiveness of the education system. 
Employers, merchants, policymakers and educators are evaluating the results of the 
educational process, but they are using different tools. It seems naive to assume that we 
would be able to assess how schools are doing with one set of methods or instruments. 
The subjects and desired results are as diverse as the institutions and communities they 
serve. If we wish to assess the effectiveness of the educational system to produce 
graduates capable of thinking critically, communicate clearly, do higher order thinking 
and problem solving. Do we have the tools, money or understanding necessary to 
measure what happens when you do systemic reform? We must be certain that the tools, 
and how they are applied, will reliably measure that which we wish to measure. 




8 



The Community College 

The community college is an interesting mix between university focus on the 
disciplines, professional/technical education, and community education. This diversity of 
mission is typically reflected in the Mission Statement of the institutions. Mt. Hood 
Community College has such a statement and it pledges to fulfill (among others) the 
following commitments: 

• provide access to technological education and training, a wide variety of transfer 
programs, developmental education and comprehensive community services. 

• strive to help students discover their own potential, respect the uniqueness of others, 
and develop ethical values. 

• provide enthusiastic and dedicated instruction to students of diverse cultural and 
economic backgrounds. 

• collaborate with agencies, organizations and businesses to best serve students and the 
community and to be responsive to the realities and demands of a changing world. 

• be a community of caring individuals who strive for excellence in all facets of college 
life. 

The essence of the mission statement outlines in social terms how the institution provides 
service to the community through education. It also defines in general terms the 
environment in which this service is going to occur, “a community of caring individuals.” 
In particular it describes the learning environment, a place with dedicated and 
enthusiastic instructors in the classrooms whose primary obligation is student 
achievement of personal goals. In the process we will provide a positive environment for 
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socialization to occur, and a system from which students will realize their own potential. 
What a positive and lofty role to fulfill in the community process. 

Are we doing it? How do we know? How can we evaluate what we do? What is it 
exactly that we want to measure? Do we have reliable instruments to measure what we 
wish to know? The list of questions relating to assessment goes on and on. Many of the 
problems being explored parallel aspects of the processes of institutional effectiveness. 
There are external and internal evaluators which can be applied when examining the 
question of institutional effectiveness. The pressure for external indicators of 
effectiveness are driven by accountability concerns which are relatively external to the 
institution. How does the state know what the institutions are doing and how can the 
institution report on its effectiveness, thus proving itself more accountable. All kinds of 
effectiveness indicators can be proposed. 

The American Association of Community Colleges identifies a core of thirteen 
indicators that make sense to the various public entities served by the community college. 
These apply primarily when used to assess effectiveness where the focus is on 
accountability issues, whether it is the Board, the State, or the Federal government. The 
concepts for consideration were not, by design, oriented to assess system effectiveness. 
The document identifies a select core group of indicators that address the issues, implicit 
and explicit, in the community college mission. The various indicators reflect the goals in 
the mission statements for nearly all community colleges; These include student progress, 
career preparation, transfer preparation, general education, customized education, and 
community development. In some areas indicators are tools for internal assessment, while 
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others focus outside the institution. Some indicators have direct application in the 
assessment of systemic reform as well as evaluation of community college effectiveness. 



1 . Student Goal Attainment 

2. Persistence (Fall to Fall) 

3. Degree Completion Rates 

4. Placement Rate in the Work Force 

5. Employer Assessment of Students 

6. Number and Rate of those Who Transfer 

7. Performance After Transfer 

8. Success in Subsequent, Related Coursework 

9. Demonstration of Critical Literacy Skills 

10. Demonstration of Citizenship Skills 

1 1 . Client Assessment of Programs and Services 

12. Responsiveness to Community Needs 

13. Participation Rate in Service Areas 

Arguably any of these Core Indicators (Cl) could be utilized in assessment of systemic 
reform. Part of the distinction lays in the level at which assessment occurs, and if the 
assessment is internally directed or external. Cl number 6 and 7, exhibit this duality. Core 
Indicator 6 is an internal examination of transfer rates, while Cl 7 is an external indicator 
of effectiveness as measured by how well the students do after they transfer. Core 
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Indicator 9 and 10 display a similar sense of intemal/extemal. Nine, addresses critical 
literacy skills (communication, logic), 10 is external and speaks to social qualities 
instilled in the individual due to participation in the experience. Those measures whose 
focus is external to the college environment may have more direct application in the 
assessment of systemic reform. 

The Students and the Faculty 

One of the faculty to whom I spoke in preparation for this paper compared 
teaching at a major university and at a community college. “Community college is 
Teaming within the two minute warning. Our customers are different, motivated by 
necessity not academics.” What should we be investigating, since we are not assessing 
academicians, and scholarship is not the issue. Rather we are assessing self-learning 
skills, lifelong learning skills (for when jobs become obsolete). Consequently mastering 
concepts/content is not as central to the process of assessment. Given background 
materials, a more relevant approach causes them to assemble and apply what is intrinsic 
to the concepts presented. Skills needed include the ability to organize facts, give relevant 
examples, demonstrate the ability to apply what is learned. Modeling the instructional 
application to the self-learning process allows development of self-learning skills rather 
than content. Relative to the consequences of community college to the student, the 
relevant outcome is strong self learning skills over the mastery of content. 

The goal becomes one of developing a background in their discipline to aid in the 
integration of information in real world settings— common sense, reading the situation. 
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gauging, making appropriate decisions. The whole issue of students’ capacity to integrate 
learned disciplines needs to be addressed in the teaching process, that is, concepts should 
not be learned or applied in isolation. 

Anatomically we look at the shape of the arm, we integrate knowledge of physics 
to better comprehend how the muscles and bones work together. Learning does not just 
equal being familiar with material content, but the ability to manipulate and reapply 
information in the context of a different setting. Life is not a multiple choice test. The 
average community college student- a thirty-two year old female, knows this. The 
comfort zone for many traditional college students is regurgitating material in terms of 
familiar concepts, but familiarity does not equal learning. As a result, assessment requires 
application of materials in context, it is important to go beyond familiarity, to probe 
understanding and connectivity. 

What are the tools, how are they applied, and will they reliably measure that 
which we wish to measure? We have gross cut measurement tools-such as college 
placement tests, student retention, performance on teacher created class and laboratory 
exams, and standardized tests. Many of these traditional tools are weak, for example, the 
problems of measuring student retention is replete with problems. Standardized exams do 
not serve as valid measurements, because they are not holistic and can be prepared for 
specifically. Further they are by design objective, not free form where someone is put into 
a situation where there are no clues other than the problem in front of them, and they 
must determine the nature of the problem, assess the relevant aspects of the information, 
remove irrelevant stuff, create hypothesis, do corrective changes, and solve the problem, 
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adapt to the problem or somehow accommodate the conditions of the problem. We have 
no quick way to assess that ability, and even we, as science professionals, can not always 
recognize that which is or is not relevant to the solution of a problem. Academic and 
professional training reinforces comfort levels with tools that measure precisely, but 
sometimes what we are measuring may have little application in the assessment of how 
the system is functioning in its more global imperative. It is possible that what we in the 
sciences really should prepare our students to do, when they exit from our programs, will 
have little to do with the particular subject being taught. Subject matter will be evaluated 
by how well it meets educational demands as interpreted in the context of both academic 
and social applications. 

Review and Further Discussions 

You’ve removed most of the roadblocks to success when you’ve learned the 

difference between motion and direction. (Bill Copeland) 

What are the risks of moving education from the profound to the pragmatic? A 
shift in systemic evaluation to value social as well as academic priorities would not make 
implicit the need to abandon traditional subjects. Academic rigor plays an important 
function in the development of discipline specific knowledge. Systemic reform might 
encourage asking how discipline specific knowledge fits in with the larger vision of 
education’s role in the development of the whole. 

In the design or architecture of educational systems one needs to recognize the 
gap that exists between what is and what should be. It can be applied to a five minute 
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presentation, or can find application on an entire educational complex or system. Need 
analysis or needs assessment is the understanding of the disparity between what is and 
what should be. What seems to be lacking in the process of systemic reform examinations 
is the what should be. There have been a number of reports which have come out which 
look at what is, but it is the examination between what is and what should be that shows 
the gap. One can talk about the need being the shortfall between the two, and then look at 
the methods analysis and develop possible strategies to reduce that gap. 

One area systemic reform needs to address is what is the source of the 
discontinuity. This gap between what is and what should be, is it because we don’t like 
what we are doing? Where is the reform leading us, and what possible values can be 
derived by the institution, society and consumers of education? Clear vision is 
consummate, one needs to ask where we are headed in reformation. If you don’t have that 
vision set in front, it is hard to talk about systemic reform, about where we should be. So 
the fundamental issue is do we agree on what the fundamental issues are, on the vision of 
what should be. Otherwise you do not get reform with purpose, but rather with agitation. 
You are just homogenizing things, stirring things around, making change for change sake. 
When we really ask why are we changing, we usually respond with, “Well, what we are 
doing currently is not working.” This is a cyclic process— I do not know the exact cycle 
but it seems to be about every twelve years or so. Often the reasons for the prescribed 
change are identical in essence to the reasons which lead to the change which brought us 
to the current situation. Somebody has to establish our vision of what should be, then we 
can set our “collective system sails” on course and move in that direction. 
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The seductive appeal of outcomes and competencies is a crystallizing of our 
perceptions of what students can perform when they are done. A performance outcome 
allows us to recognize the accomplishments of our applications, and on a higher level we 
may want the student to deduce what processes are occurring. So in science, rather than 
do a titration process, we may want to instill the understanding of what is the process for 
neutralizing hazardous materials, and how one can safely handle chemical materials. How 
you phrase your outcomes will ultimately determine the type of reform you want to put 
into the system. But the people positing the outcomes are in the midst of the system, so 
what are the chances that the proposed reforms will look much different from what we 
already have? How would chemistry look different in somebody’s proposal for systemic 
change? 

Who determines the outcomes, is it the market place, the consumer of education, 
or the teacher making a course outline? Who are the shareholders? What if proposed 
outcomes were negotiated by societal participants? Teachers should not be the ones 
engineering educational reform. It should be the society and all the shareholders. What if 
our primary outcome of the educational system was that graduates, when they left the 
institution, did not negatively impact society’s social system? Or that successful 
graduates from the system were measured by societal interaction to be self-sufficient, 
self-supporting, able to interact with others in the free enterprise system, and cognizant of 
the role of the constitution in everyday life. Reform is accomplished when graduates are 
equipped with the skills necessary to get a job and make a positive contribution to 
American or World society. These are not terms which educators are typically 
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comfortable with using, and the processes for measuring this type of outcome are even 
more unfamiliar. But that is typically what parents say, what civic leaders say and what is 
often unwritten as a value indicator of an educated individual. How or where does this tie 
into education reform under the current paradigm of the educational system? 

This is truly a different set of priorities than the traditional thought that an 
educated individual can read, write, do basic math or the like. So who are the 
shareholders of this different set which begs the question of who should be involved in 
the discussion about what are the appropriate outcome for education. Should it be 
scientist, or scientific leaders? Certainly they are shareholders, but who else needs to be 
included in the discussion? How do we in the process of systemic change allow for those 
voices to be heard, and be answered? Educational institutions have traditionally been one 
of the more difficult establishments in which to bring about change. This is possibly 
because of the self-perceived role of teachers in the process of what is or is not 
appropriate outcomes of their classroom and summarily the educational system. They 
may not wish to phrase it this way but, in essence, it is their perceived obligation/right to 
have control over the content and outcomes of the classes, what will be said or not said, 
what will be taught or emphasized within the parameters of the course outline. So who 
truly posits what the outcomes will be? Are we thinking radically enough, and are our 
thoughts far reaching? 

If we are truly trying to move to a more holistic approach to the education of 
students, then why do we still divide into camps? Why do we still have disciplines such 
as science, or mathematics? These are traditional development areas. The difficulty is that 
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this image limits peoples willingness to cross discipline lines. You have to be in a science 
class to think about science, or physics or whatever. How do you ever get to the point 
where you can ask, what was it that you just did, and how does that apply in your life, 
outside of the class in which you are currently? 

What if we shift the outcomes and how they are presented and measured. For 
example, let’s use an entry level computer literacy course. Like most entry level courses, 
whether in computers, math, chemistry or what have you, one is typically going to be 
introduced to a new vocabulary. Framework concepts or schema are going to be 
developed upon which one can begin to hang information. This then leads to “cyber- 
anatomy” becoming the primary focus of the class, and most of the exercises are 
developed with this concept in mind. But what are the functional outcomes that a student- 
-educated consumer/computer literate individual, needs? Suppose we change the outcome 
by trying to develop a relevant exercise to use the information to do certain things, such 
as access information off the Internet, do word-processing, learn new programs, exhibit 
operational skills. What the new outcome really desires is functioning consumers, able to 
buy their own computer system, to solve the problems that they deem appropriate for a 
computer. If they want to start their own business, keep the books, and use a computer, 
they will be able to select the software and the hardware necessary, and seek the 
appropriate training. The new outcome, is that they become good consumers. The over- 
riding outcome is that all the other stuff about vocabulary, which might have been 
traditionally the course outcomes, becomes tools to achieve the skill level of educated 
consumers. The vocabulary, an essential tool, is pertinent to reading the advertising, 
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ascertaining what kind of software they want, what it can or cannot do, and how it might 
be used in their lives—home, work or other environment. 

Suddenly things are flipped. No longer is an outcome naming the five parts of a 
computer, and the ability to list and define vocabulary. Likewise, no longer can 
chemistry use traditional outcomes such as naming the inert elements or the ability to 
calculate moles, to measure appropriate grasp of the subject. Biologists can no longer 
measure achievement by the learned definition of a genus, Species, or gamete. Now the 
emphasis switches to application in context. The students from computer science 
recognizes that they are going to be doing a lot of typing; from anatomy they recognize 
the shape and function of the bones of the wrist and hand, neck or eyes. As a consumer 
they are going to be concerned about ergonomics, screen size or shape, health effects of 
computers. Quickly the emphasis and application of education transforms or is in essence, 
reformed. This then elucidates what the elements of systemic reform become as its 
participants change from teachers to consumers, and as desired outcomes metamorphose 
from micro to macro. 

The emphasis is that it is what we want people to be able to do when they are 
done with what the system provides. What skills do we want them to take away from the 
experience? Critical to this issue is the delivery system which we are creating for various 
developmental areas of educational distance learning. How will we do virtual education 
through the Internet, and how will we measure our success? We are planning to use 
lecture-based and lab-based tools to measure an experience that we have not even 
experienced yet, and for many teachers, one which they never will experience. How do 
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we know that we are not using the wrong tools to assess results, or that they will be 
accurate measurements? Are we measuring volumes of water with a yard stick? This 
inevitably occurs with outcome assessment— what they say is that they want six inches of 
water, but water is three dimensional. This is the kind of conundrum we face when we 
look at systemic reform. What we are really talking about is a paradigm shift, and 
education has never experienced any radical paradigm shifts. We are still in these 
buildings, are we not? We still have lecture halls, classrooms, laboratories facilities, and 
we have had those for centuries. This is probably not going to disappear with the advent 
of the computer and Internet. 

So what is driving this reform? Is it just that we have reached that point in the 
cycle, is it a matter of accountability? I once heard a teacher say that they liked what they 
did because it was impossible to measure how successful they were, and that there is no 
way to prove or disprove what they had done. Not like being a carpenter, where at the end 
of the day you have tangible evidence of your accomplishments. We seem to have two 
major (simplistic) approaches to education, the IQ approach and the mastery approach. 
The community college wrestles with both tensions. The traditional lower division 
transfer area dictates that we subscribe to processes of evaluation similar to those done at 
the transfer institutions. Consequently rigor and testing must be comparable. We have 
the professional/technical side where people have to perform and have the skill base. This 
is a very sophisticated level. Training is ranges from remedial to high plane, but it is also 
pragmatic. “I have to be able to do this or that”, and the goal is measurable. 
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This then begs the question, do we really believe in change? Who are we as 
educators to initiate change, are we not the products of the old paradigm? Should we be 
in control of the rudder, are we as educators the best qualified to direct where we should 
head? Those who have come through the system have vested interest in the survival of 
the system as it is. If radical change occurs, then what does that do to the credibility of 
their credentials? We know from experience that when it comes to paradigm shifts, those 
most vested in the existing model are least likely to move the change forward, even if 
they created the new paradigm. An interesting and well known example of this is the 
Swiss watchmakers who came up with the idea of the quartz crystal time pieces. They 
discovered and developed the concept of the quartz crystal for keeping time but, because 
they were unwilling to move into a new way of thinking about watches, they lost the lead 
and eventually ninety percent of the market share. 

Has education in recent times experienced a paradigm shift, and if so what was it. 

I find it difficult to answer this question, especially if you think of a paradigm shift being 
something in the magnitude of a major concept being replaced totally by another-round 
Earth/flat Earth. Even replacing teachers, in the context of re-placing them in the 
educational process, does not meet the requirements of a paradigm shift. Moving teachers 
from the position of information distributors, to coaches, facilitators and evaluators may 
not be adequate to qualify. What about distance learning? Here the shift is from a single 
room with many people interacting in socially condoned environments to many people in 
different rooms interacting in a non-social environment. No longer do we teach, learn and 
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share in a person to person, face to face situation. Social norms are not conveyed, social 
consequences of dress, manners and the like are non-existent. 

At what level are we going to work, relative to systemic reform? Generally people 
are willing to agree at the broadest level, but unwilling at specific levels. When we 
attempt to assess training it is at four levels: 

1 . did they like it 

2. could they do it 

3. could they do it away from the classroom environment (on the job). 

4. did it solve the problem 

By and large training assessment has focused on number one. Did they like what they 
experienced? Would you recommend it to your friends? This is basically an attitudinal 
measure of satisfaction. Assessing ability to carry out a task in a controlled environment 
was also typically measured, but whether there was transference to the job was generally 
ignored. But the ultimate test is most probably the last, did it solve the problem, which 
broaches the question, why did you have them do it in the first place? 

So why then do we have students do mathematics, why take science? It is possibly 
because we want people to be generalists, life long learners, sifters of information. The 
wide knowledge base allows the individual to be placed in new situations, or if something 
new appears they are able to draw on this knowledge. This enables the individual to sift 
the information and derive from that analysis an estimate of what is relevant. What we 
need to do is ascertain the solution to the problems. When people say that we need people 
to be able to read and write, is that really the problem that they are addressing, does it 
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really solve the problem? What impact does it have that they can not read and write? 
When we use the word need as a verb we are not necessarily addressing the solution to 
the problem. People often jump to the solution before assessing the real problem. 

So what is education reform? Do we really know where we are now? Do we have 
a vision of what it should be, what it should look like, and how to get there? What are the 
desired outcomes of the process of education? Once these points are clearly configured, 
we should be able to develop the process by which we can assess how we are doing. 
Without a crystallized image of what exactly we are trying to do, we will continue to try 
to measure volumes of water using a yard stick. 

Performance curves tell us that more and more energy is required to make 
improvements to a system as it approaches completion. That is, eighty percent of the 
work gets done on a project with 20 percent of the time and effort, and the last 20 percent 
to completion takes 80 percent of the effort and time. Is this the case with our educational 
system, are we in the upper percent of our performance curve? Are there no more minor 
“tweaks” that will make any significant difference? Can we really see any minor 
adjustments that we can make? Possibly the answer is no. Can we test better, have better 
classrooms, better equipment.. ..probably. Will it make any significant difference in the 
outcome of the system... probably not. Are we just polishing brass on the Titanic? 
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